NOTE: The original version of this post appeared HERE. Cross-posted here per suggestion of @Vitaliy Kaurov

I feel this is a good opportunity to list some error-checking techniques. I will discuss those I'm aware of, and please feel free to edit this post and add more. I think the main questions to answer here are what we would like a function to return in the case of error, and how to do this technically.

What to return on error

I can see 3 different alternatives here

The function issues an error message and returns unevaluated.

This one is more appropriate in the symbolic environment, and corresponds to the semantics that most Mathematica functions have w.r.t. errors. The idea is that the symbolic environment is much more forgiving than more traditional ones, and by returning the same expression we indicate that Mathematica simply does not know what to do with this. This leaves a chance for the function call to be executed later, for example when some of the symbolic arguments acquire numeric or other values.

Technically, this can be achieved by exploiting the semantics of conditional patterns. Here is a simple example:

ClearAll[f];
f::badargs = "A single argument of type integer was expected";
f[x_Integer] := x^2;
f[args___] := "nothing" /; Message[f::badargs]

f[2]

4

f[1, 2]

f::badargs: A single argument of type integer was expected

f[1, 2]  

The idea is that at the end, the pattern is considered not matched (since the test in condition does not evaluate to explicit True), but the Message gets called in the process. This trick can be used also with multiple definitions - since the pattern at the end is considered not matched, the pattern-matcher goes on testing other rules down the DownValues list. This can be either desirable or not, depending on the circumstances.

The nature of the function is such that it is more appropriate to return $Failed (explicit failure).

Typical examples of cases when this may be appropriate are failures to say write to a file or find a file on disk. Generally, I'd argue that this behavior is most appropriate for functions used for software engineering (in other words those functions that don't pipe their results straight into another function but call other functions which should return, and form the execution stack). One should return $Failed (possibly also issuing an error message) when it does not make sense to continue execution once the failure of a given function has been established.

Returning $Failed is also useful as a preventive measure against the occasional regression bugs resulting from implementation changes, for example when some function has been refactored to accept or return different number and/or types of arguments, but the function to call it has not been promptly updated. In strongly-typed languages like Java, the compiler would catch this class of errors. In Mathematica, this is the programmer's task. For some inner functions inside packages, returning $Failed seems more appropriate in such cases than issuing error messages and returning unevaluated. Also, in practice, it is much easier - very few people would supply error messages to all their inner functions (which also is probably a bad idea anyway, since the user should not be concerned with some internal problems of your code), while returning $Failed is fast and straightforward. When many helper functions return $Failed rather than keep silence, the debugging is much easier.

Technically, the simplest way is to return $Failed explicitly from within the body of the function, using Return, such as in this example custom file-importing function:

ClearAll[importFile];
Options[importFile] = {ImportDirectory :> "C:\\Temp"};
importFile::nofile = "File `1` was not found during import";
importFile[filename_String, opts : OptionsPattern[]] :=
 Module[{fullName = 
     getFullFileName[OptionValue[ImportDirectory], filename], result},
   result = Quiet@Import[fullName, "Text"];
   If[result === $Failed,
      Message[importFile::nofile, Style[fullName, Red]];
      Return[$Failed],
      (* else *)
      result
]];

However, very often it is more convenient to use the pattern-matcher, in a way outlined in the answer of @Verbeia. This is easiest for the case of invalid input arguments. For example, we could easily add a catch-all rule to the above function like so:

importFile[___] := (Message[importFile::badargs]; $Failed)

There are more interesting ways to use the pattern-matcher, see below.

The last comment here is that one problem with chaining functions each of which may return $Failed is that lots of boilerplate code of the type If[f[arg]===$Failed, Return[$Failed],do-somthing] is needed. I ended up using this higher-level function to address this problem:

chainIfNotFailed[funs_List, expr_] :=
 Module[{failException},
  Catch[
   Fold[
    If[#1 === $Failed,
      Throw[$Failed, failException],
      #2[#1]] &,
    expr,
    funs], failException]];

It stops the execution via exception and returns $Failed as soon as any intermediate function call results in $Failed. For example:

chainIfNotFailed[{Cos, #^2 &, Sin}, x]

Sin[Cos[x]^2]

chainIfNotFailed[{Cos, $Failed &, Sin}, x]

$Failed

Instead of returning $Failed, one can throw an exception, using Throw.

This method is IMO almost never appropriate for the top-level functions that are exposed to the user. Mathematica exceptions are not checked (in the sense of say checked exceptions in Java), and mma is not strongly typed, so there is no good language-supported way to tell the user that in some event exception may be thrown. However, it may be very useful for inner functions in a package. Here is a toy example:

ClearAll[ff, gg, hh, failTag];
hh::fail = "The function failed. The failure occured in function `1` ";

ff[x_Integer] := x^2 + 1;
ff[args___] := Throw[$Failed, failTag[ff]];

gg[x_?EvenQ] := x/2;
gg[args___] := Throw[$Failed, failTag[gg]];

hh[args__] :=
  Module[{result},
   Catch[result = 
     gg[ff[args]], _failTag, (Message[hh::fail, Style[First@#2, Red]];
      #1) &]];

and some example of use:

 hh[1]

 1

hh[2]

hh::fail: The function failed.
The failure occured in function gg 
$Failed

hh[1,3]

hh::fail: The function failed. 
The failure occured in function ff 
$Failed

I found this technique very useful, because when used consistently, it allows to locate the source of error very quickly. This is especially useful when using the code after a few months, when you no longer remember all details.

What NOT to return

• Do not return Null. This is ambiguous, since Null may be a meaningful output for some function, not necessarily an error.

• Do not return an error message printed using Print (thereby returning Null).

• Do not return Message[f::name] (returning Null again).

• While in principle I can imagine that one may wish to return some number of various "return codes" corresponding to different types of errors (something like enum type in C or Java), in practice I never needed that in mma (may be, it's just me. But at the same time, I used that a lot in C and Java). My guess is that this becomes more beneficial in more strongly (and perhaps also statically) typed languages.

Using the pattern-matcher to simplify the error-handling code

One of the main mechanisms has been already described in the answer by @Verbeia - use the relative generality of the patterns. With regards to this, I can point to e.g. this package where I used this technique a lot, as an additional source for working examples of this technique.

The multiple message problem

The technique itself can be used for all of the 3 return cases discussed above. However, for the first case of returning the function unevaluated, there are a few subtleties. One is that, if you have multiple error messages for patterns that "overlap", you'd probably like to "short-circuit" the match failure. I will illustrate the problem, borrowing the discussion from here. Consider a function:

ClearAll[foo]
foo::toolong = "List is too long";
foo::nolist = "First argument is not a list";
foo::nargs = "foo called with `1` argument(s); 2 expected";
foo[x_List /; Length[x] < 3, y_] := {#, y} & /@ x
foo[x_List, y_] /; Message[foo::toolong] = Null
foo[x_, y_] /; Message[foo::nolist] = Null
foo[x___] /; Message[foo::nargs, Length[{x}]] = Null

We call it incorrectly:

foo[{1,2,3},3]

foo::toolong: List is too long
foo::nolist: First argument is not a list
foo::nargs: foo called with 2 argument(s); 2 expected
foo[{1,2,3},3]

Obviously the resulting messages are conflicting and not what we'd like. The reason is that, since in this method the error-checking rules are considered not matched, the pattern-matcher goes on and may try more than one error-checking rule, if the patterns are constructed not too carefully. One way to avoid this is to carefully construct the patterns so that they don't overlap (are mutually exclusive). A few other ways out are discussed in the mentioned thread. I just wanted to draw the attention to this situation. Note that this is not a problem when returning $Failed explicitly, or throwing an exception.

Using Module, Block and With with shared local variables

This technique is based on the semantics of definitions with conditional patterns, involving scoping constructs Module, Block or With. It is mentioned here. A big advantage of this construct type is that it allows one to perform some computation and only then, somewhere in the middle of the function evaluation, establish the fact of the error. Nevertheless, the pattern-matcher will interpret it as if the pattern was not matched, and go on with other rules, as if no evaluation of the body for this rule had ever happened (that is, if you did not introduce side effects) . Here is an example of a function that finds a "short name" of a file, but checks that a file belongs to a given directory (the negative on which is considered a failure):

isHead[h_List, x_List] := SameQ[h, Take[x, Length[h]]];

shortName::incns = "The file `2` is not in the directory `1`";
shortName[root_String, file_String] :=
  With[{fsplit = FileNameSplit[file], rsplit = FileNameSplit[root]},
    FileNameJoin[Drop[fsplit, Length[rsplit]]] /;isHead[rsplit, fsplit]];

shortName[root_String, file_String]:= ""/;Message[shortName::incns,root,file];

shortName[___] := Throw[$Failed,shortName];

(In the context where I use it, it was appropriate to Throw an exception). I feel that this is a very powerful technique, and use it a lot. In this thread, I gave a few more pointers to examples of its use that I am aware of.

Functions with options

The case of functions receiving options is IMO not very special, in the sense that anything I said so far applies to them as well. One thing which is hard is to error-check passed options. I made an attempt to automate this process with the packages CheckOptions and PackageOptionChecks (which can be found here). I use those from time to time, but can not say how of whether those can be useful for others.

Meta-programming and automation

You may have noticed that lots of error-checking code is repetitive (boilerplate code). A natural thing to do seems to try automating the process of making error-checking definitions. I will give one example to illustrate the power of mma meta-programming by automating the error-checking for a toy example with internal exceptions discussed above.

Here are the functions that will automate the process:

ClearAll[setConsistencyChecks];
Attributes[setConsistencyChecks] = {Listable};
setConsistencyChecks[function_Symbol, failTag_] :=
    function[___] := Throw[$Failed, failTag[function]];


ClearAll[catchInternalError];
Attributes[catchInternalError] = {HoldAll};
catchInternalError[code_, f_, failTag_] :=
  Catch[code, _failTag,
    Function[{value, tag},
      f::interr =  "The function failed due to an internal error. The failure \
           occured in function `1` ";
      Message[f::interr, Style[First@tag, Red]];
      f::interr =.;
      value]]; 

This is how our previous example would be re-written:

ClearAll[ff, gg, hh];
Module[{failTag},
  ff[x_Integer] := x^2 + 1;
  gg[x_?EvenQ] := x/2;
  hh[args__] := catchInternalError[gg[ff[args]], hh, failTag];
  setConsistencyChecks[{ff, gg}, failTag]
];

You can see that it is now much more compact, and we can focus on the logic, rather than be distracted by the error-checking or other book-keeping details. The added advantage is that we could use the Module- generated symbol as a tag, thus encapsulating it (not exposing to the top level). Here are the test cases:

hh[1]

 1

hh[2]

 hh::interr: The function failed due to an internal error.
 The failure occured in function gg 
 $Failed

hh[1,3]

hh::interr: The function failed due to an internal error. 
The failure occured in function ff 
$Failed  

Many error-checking and error-reporting tasks may be automated in a similar fashion. In his second post here, @WReach discussed similar tools.