You'll need to check the
list of the countries that have the visa exemption agreement with Japan.
If your country is included in the list, then you will not be required for a
visa when entering Japan under normal circumstances. If your country is not
included in the list, you'll need to obtain a short-term stay visa from the
embassy of Japan in your country.
Please read the Japanese visa guide provided by the ministry of foreign affairs carefully. And be informed that it is the embassy of Japan in your country that makes a decision on issuing a visa. Whether the attendees need Letter of Invitation fully depends on the relations between countries. The conference organizers hereby reiterate that we are neither able to nor authorized to decide the relations between countries.
For more details, please visit: http://www.mofa.go.jp/j_info/visit/visa/index.html.