What is the dividing line between literary fiction and the fantastical? Is there a line? Is there also a distinction between fantasy and science fiction? Are these even meaningful questions?
In thinking about this and discussing it in the past, I’ve tended to prefer the terms used by Moorcock and Le Guin in their literary essays, ‘realist’ and ‘imaginative’ fiction if a dividing line must be drawn. Realist fiction tends to favour close simulation of ‘realistic’ human experience, and is most highly characterised (I think) in modern fiction by stream of consciousness narratives about ordinary things. Imaginative fiction tends to lean towards an exploration of the possibilities of human experience, and both fantasy and science fiction tend to be more imaginative than realist in bent.
As for defining SF and fantasy as terms, I’m going to agree with Karl Popper that there are two ways (I am avoiding the word ‘valid’) to arrive at definitions.
The first is a common usage definition. This is a dictionary definition, by which we ask ‘what do people generally mean by this?’ This is easier if the term is ‘puppy’, harder to pin down if the term is debatable among people (and very hard if the term is a cultural artefact, because by definition it does not physically exist: what is meant by the colour ‘blue’ varies among societies. Some cultures don’t distinguish between green and blue, and some people in cultures that do distinguish between green and blue will define aquamarine as blue, whilst others will call it green: it is intrinsically hopeless to attempt ‘definitive’ or ‘all-encompassing’ definitions of cultural artefacts in this way).
By attempting a common usage definition we may arrive at a set of multiple and even contradictory definitions, and they are all of them correct, because we accepted at the beginning we are merely trying to define what is meant by people who use the term, and the populace consists of more than one person. Perspectives and experiences are not identical.
The other way to define a thing, is what Popper would have said is more scientific, or more relevant to evidence-based approaches to knowledge about the physical world, and this is to define things from right-to-left, not left-to-right. So, In this view, it is inexact or not useful to ask ‘what is the essential nature of being a puppy? What is puppyhood?’ Rather, we have identified that there is such as thing as a young dog. This thing seems to have different behaviour and some differences in physiology and morphology to adult dogs. It would be useful to have a term for this thing. Let us call it a ‘puppy’.
So, for example, the common usage definition of ‘chemical’ would be something like ‘a highly refined and often artificial substance’. When a company claims their food product has ‘No added chemicals’ they mean this. Chemists of course, will be irritated because they established the word ‘chemical’ in a right-to-left fashion, and this meaning is quite different.
Right-to-left chemicals: It appears that all matter is made up of elements. These elements can exist as pure elements and as compounds with other elements in a pure forms. It would be useful to have a term for these pure substances. Let us call them ‘chemicals’. By this definition all matter is a chemical, including water.
So that, by this definition the ‘No added chemicals’ label is nonsense, but getting wrought up about it is not very practical either. It would involve a confusion about the underlying modes of definition.
Anyway, on with fantasy and science fiction. There is clearly a difference in common usage between these terms, but (as is obvious every time this is debated at any con anywhere), the common usage definitions differ enough among readers and writers that the argument is more or less unresolvable (i.e. what do you mean that object is blue? It’s clearly green!).
The personal definition I give people when pressed is that I tend to think that fantasy is imaginative fiction that deals with imaginative elements that are primarily not plausible given the current scientific understanding, and that science fiction is imaginative fiction that deals with elements that are primarily plausible given the current scientific understanding. Using this definition, Star Wars is (mostly) fantasy, whereas the Pern novels are (mostly) science fiction (although both are a blend of the two).
I suppose you could take this further and argue that hard fantasy is knowingly and self-consciously unconcerned about explaining the imaginative element(s) in any scientific way (Kafka comes to mind), whereas hard science fiction is self-consciously and knowingly concerned about explaining the imaginative elements in a scientific way (Jules Verne and his descendants). I’m unconvinced either term is very useful though.
In the end, as Vonnegut said, ‘all great fiction is about what a bummer it is to be human’. I am less cynical perhaps, and sometimes I will commit the unpardonable crime of modifying Vonnegut to ‘all great fiction is about what it is to be human’, although it depends on the day.
Both F and SF do this in ways that are identifiably the same (i.e. either might address mythic journeys and magico-religious experiences), but so too do other genres, horror, nurse fiction, westerns, pulp romance and lit-fic. To use a couple obvious SFF examples, the Ravaging of the Shire is an examination of a theme more usually found in SF (dehumanization / modernization / industrialization), whereas 2001 has at least one thematic element more usually associated with fantasy (the magico-religious experience of enlightenment).