To investigate if top-down contingent capture by color cues relies on verbal or semantic templates, we combined different stimuli representing colors physically or semantically in six contingent-capture experiments. In contingent capture, only cues that match the top-down search templates lead to validity effects (shorter search times and fewer errors for validly than for invalidly cued targets) resulting from attentional capture by the cue. We compared validity effects of color cues and color-word cues in top-down search for color targets (Experiment 1a) and color-word targets (Experiment 2). We also compared validity effects of color cues and color-associated symbolic cues during search for color targets (Experiment 1b) and of color-word cues during search for both color and color-word targets (Experiment 3). Only cues of the same stimulus category as the target (either color or color-word cues) captured attention. This makes it unlikely that color search is based on verbal or semantic search templates. Additionally, the validity effect of matching color-word cues during search for color-word targets was neither changed by cue-target graphic (font) similarity versus dissimilarity (Experiment 4) nor by articulatory suppression (Experiment 5). These results suggested either a phonological long-term memory template or an orthographically mediated effect of the color-word cues during search for color-words. Altogether, our findings are in line with a pronounced role of color-based templates during contingent capture by color and do not support semantic or verbal influences in this situation.