a) Proving NOPREFIX(L) is regular :
The idea is simple , for a string w ∈ NOPREFIX(L) , w must be in L , and no proper prefix of w is in L
So , if we have a DFA M for L , we run string w on M , changing M a bit , if M enters the accept state before the end of the input , then a proper prefix of w is accepted by L and so we reject , if not and we reach an accept state just by the end input we accept , of course if by the end of w we don't enter an accept state we reject since w is not in L an so not in NOPREFIX(L)
Now you may want to try constructing M' based on the above explanation , then check your answer
Proof :
Let M = (Q,Σ,δ,q0,F) recognising some regular language L ,construct M' = (Q',Σ,δ',q0',F') to recognise NOPREFIX(L) as follows :
Q' = Q
for r ∈ Q' , a ∈ Σ
δ'(r,a) = δ(r,a) , if r is not in F
δ'(r,a) = ∅ , if r ∈ F
q0' = q0'
F' = F
Hopefully you can see how changes in δ' achieve our purpose , M' simulates M , till it reaches an accept state , if the input ends we accept ( note that w is not a proper prefix of w) since w ∈ L , and we did not encounter any accepted proper prefixes , but if we enter an accept state and the input is not yet finished we reject since this means that we encountered an accepted proper prefix
b) Proving NOEXTEND(L) is regular
This is an easier one , for a string w ∈ NOEXTEND(L) , we need to make sure that no string wx ∈ L , x ∈ Σ^+ , so for w not to be a proper prefix of a string in L ,we need to make sure that no symbols could be added to w to form a string in L ,
Once again you may want to try M' that recognises NOEXTEND(L)
Proof :
Let M = (Q,Σ,δ,q0,F) recognising some regular language L ,construct M' = (Q',Σ,δ',q0',F') to recognise NOEXTEND(L) as follows :
Q' = Q
δ' = δ
q0' = q0'
F' = { q| there is no path from q to an accept state in F)
M' works exactly as M , when the string end we consider the final state we reached q , if there is a path from q to any accept state in F ( the accept states of M) , then we reject since this means we can extend w by x to reach a string wx ∈ L , with x being the string of symbols on the path from q to accept state , and thus w is a proper prefix of some string wx ∈ L , if there is no path from q to an accept state so w cannot be extended to form a string in L and thus we accept