I assuming that you are able to bring up a stylesheet that you wish to edit.
In one of my stylesheets I have a modified style for Section. I can't copy out what is displayed in the stylesheet but it says:
Local definition for style "Section" and then below that shows what the Section header actually looks like.
You could copy this down into your stylesheet using the "Choose a style" dropdown menu at the top. Or you could type "Section" in the input field at the top. Or you could copy and paste it from the Default stylesheet. Then select the Section cell bracket on the right and use Ctrl+Shift+E. That shows the underlying expression for the Section style. In my case it is:
Cell[StyleData["Section"],
CellFrame->False,
ShowGroupOpener->True,
AutoItalicWords->{"Presentations", "Mathematica", "Tensorial"},
FontFamily->"Book Antiqua",
FontSize->22,
FontWeight->"Plain",
FontSlant->"Plain",
FontVariations->{"StrikeThrough"->False,
"Underline"->False},
FontColor->GrayLevel[0]]
(This is just plain text typed into the cell and the contents may be different in your case.)
The style is defined by the various options. Not all of these are defined on the Format menu, which is why I prefer to edit the underlying expression. You only have to add the things you want changed. The option you are most interested in is "CellFrame -> False". After you make the changes you want, use Ctrl+Shift+E again to toggle back to the normal display. Any frame on the Section cell should now be gone. If you have a notebook open that is using the stylesheet it should also be gone there.
If the stylesheet is attached to the notebook, then you are done and you can close it. If this is a style sheet to be used by a number of notebooks you will have to either save it, using SaveAs, in the Stylesheets folder in an application, or use the "Install Stylesheet" button at the top save it in a system location.
