Search engines use 'robots.txt', a file at the root of a website, to understand which parts of the site should not be crawled. This file contains rules for web crawlers, indicating allowed and disallowed paths for indexing. Rules in 'robots.txt' can be specific to user-agents (type of web crawlers), and they can specify which directories or files are off-limits. However, adherence to 'robots.txt' is voluntary on the part of the crawler, and while reputable search engines comply with these directives, it does not prevent all crawlers from accessing parts of the site.

Meta-tags provide metadata about the HTML document that is not displayed on the page but is processed by web crawlers. They can influence the indexing process by providing information about the page's content, indicating which keywords represent the page's content, describing the page, or instructing crawlers on which areas of the site should or should not be indexed. This helps search engines understand the context and relevance of pages. However, over-reliance on meta-tags alone is not advisable as search engines also analyse the content of the page itself to ensure the tags accurately represent the content.

Search engines employ a variety of measures to prevent spam and malicious activities from appearing in their indexed results. They utilise advanced algorithms to detect spammy or malicious behaviour, such as keyword stuffing, cloaking, or the use of malware. They also rely on manual reviews and user reports to identify and remove such content. Moreover, search engines regularly update their algorithms to respond to new types of spam and security threats. Penalties for websites engaging in these activities can include lowering their rank or removing them from the index entirely. Additionally, search engines collaborate with cybersecurity experts and other platforms to enhance their detection capabilities.

Search engines have evolved to handle dynamic content by executing JavaScript and similar languages to render pages much like a browser would. This allows the crawlers to 'see' content that is generated dynamically, ensuring that the search engine's index is as comprehensive as possible. However, it's more challenging to index dynamic content compared to static content due to the complexity of execution and rendering processes. Consequently, search engines may prioritise the indexing of static content and rely on sitemaps and additional hints provided by webmasters to better understand dynamic content. Moreover, they might not index content that requires user interaction to be displayed.

Yes, a website can be penalised, albeit indirectly, for not being crawler-friendly. If a web crawler cannot efficiently navigate a website due to poor structure, lack of a sitemap, or extensive use of non-indexable content (like images or Flash), it may not be indexed correctly. This results in a lower ranking or even omission from search results. Search engines value user experience highly, and part of that experience is delivering relevant content quickly. Websites that hinder a crawler's ability to perform its task may not be ranked as highly as those that are more accessible and easier to index.