Well that was quick... I used my intuition and was able to accomplish it by doing the following:

In mytheme.theme:

/**
 * @param $suggestions
 * @param array $variables
 */
function mytheme_theme_suggestions_input_alter(&$suggestions, array $variables) {
  $element = $variables['element'];

  if (isset($element['#attributes']['data-twig-suggestion'])) {
    $suggestions[] = 'input__' . $element['#type'] . '__' . $element['#attributes']['data-twig-suggestion'];
  }
}

/**
 * @param $form
 * @param \Drupal\Core\Form\FormStateInterface $form_state
 * @param $form_id
 */
function mytheme_form_alter(&$form, FormStateInterface $form_state, $form_id) {
  if ($form['#id'] == 'views-exposed-form-search-results') {  
    $form['actions']['submit']['#attributes']['data-twig-suggestion'] = 'search_results_submit';
    $form['actions']['submit']['#attributes']['class'][] = 'search-box__button';
  }
}

In input--submit--search-results-submit.html.twig

{#
/**
 * @file
 * Theme override for an 'input' #type form element.
 *
 * Available variables:
 * - attributes: A list of HTML attributes for the input element.
 * - children: Optional additional rendered elements.
 *
 * @see template_preprocess_input()
 */
#}
<button{{ attributes }}>
  <svg class="icon search-box__open">
    <use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/themes/custom/mytheme/build/img/svg-sprite.svg#icon-search"></use>
  </svg>
</button>

Form still submits and functions as expected. Is this the general approach or at least acceptable? I will go with this unless there is a better way.

• Looks good. Don't think there is a twig only solution, so you need to do something like this. Theme suggestion like you did or one big input.html.twig with all input versions which you select by the attributes you set in form alter. – 4k4 Jan 24 '17 at 17:16                                    
• I'll probably want to use a data-attribute on the tag instead of just the name, most likely. – Kevin Jan 24 '17 at 17:23                                    
• Yep - I changed it to a data attribute so it doesn't interfere with anything else that may be relying on the name attribute. Works great. – Kevin Jan 24 '17 at 18:01                                    

You simply could use inline_template.

$build['hello']  = [
  '#type' => 'inline_template',
  '#template' => '<button type="button" class="search-box__button">
     <svg class="icon search-box__open">
       <use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/build/img/svg-sprite.svg#icon-search"></use>
     </svg>
   </button>',
];

• Won't you lose anything added on by {{ attributes }}? – Kevin Mar 1 '17 at 13:38                                    
• I guess you would have to set your own attributes in the '#context' array. – leymannx Mar 1 '17 at 13:41