When editing the selected model you can either: edit at the cell scope (what you're OP is mentioning) or edit at the notebook scope
To edit it for the selected notebook, you want to edit the model selection through the toolbar.
There's a pull request to make this selection a bit more intuitive globally (AKA a default state for any session) here: https://github.com/WolframResearch/Chatbook/pull/304
Before that gets merged and pushed out to users, there's a programmatic way you could set this so that any new chat notebook uses gpt-4
CurrentValue[ $FrontEnd, { TaggingRules, "ChatNotebookSettings", "Model" } ] = "gpt-4”
