Look in common bot hideouts. Yew trees, because their logs are so valuable, are very common trees for bots to be cutting. However, bots can also be found catching fish in Draynor Village, fishing for lobsters, or mining at the Crafting Guild. Bots can also be found picking flax and spinning it in Seer's Village and cutting down the magic tree's in Seer's village. The green dragons west of Daemonheim or West of Edgeville are also popular spots. The blue dragons in Taverly dungeon are also good spots. Many bots can often be found teleporting burthrope while runecrafting. They can also occasionally be found in the Dwarven Mines.
Look at their user name. Does it seem computer-generated? Is it just random letters and numbers? Some botting programs change the username, and since it is software, it has little creativity. This step should just raise your suspicion, and you should not take any immediate action until the player is confirmed as a bot.
Do they respond? Most the time a bot will not respond to your trade request or little "hi!" because no one is at the computer. Most Botters will bot during the hours of school/work or during the night 10pm-10am. Some modern bots can talk, however. See if they respond suspiciously, such as responding to the same question twice in the exact same way, or responding immediately.
Ask a trick question. If they're mining, ask them their woodcutting level. If they're woodcutting, ask them their smithing level. If they respond with the skill that they are currently working on, they're either bots or are easily-tricked humans.
What time is it?. Most Botters will bot during the hours of school/work or during the night 10pm-10am.
Follow them around. Do they do the exact same thing over and over? Do they respond immediately when the tree falls or the rock disappears? Do they acknowledge the fact that you're following them? Do they click on the exact same spots when banking? Plant marker seeds(Faruq in Al Kharid sells them)on the spots where they click to see if it disorients them.
If you are certain it is a bot, report them! Almost all bots either allow a player to gain large amounts of money easily, which ruins the economy or gives them an unfair way to achieve levels regular players work for months to get.