Christopher Bedford
Data Lackey Labs
Chris is a Databricks Certified Spark Developer and frequent blogger at http://datalackey.com
Silicon Valley Code Camp : October 19 & 20, 2019Session
Time Travails With Java, Scala and Apache Spark
After a quick refresher of Unix epoch-based time keeping, we will explore time and date processing functions in Spark SQL, and the particular problems associated with making time window-based aggregations work as expected. (Advanced)
About This Session
With leap years and varying rules by Locale, time processing has proven tricky to get right on the JVM platform. This session will provide a very quick refresher of Unix epoch-based time keeping, time string formats and Locales, and then move to a discussion of time and date processing functions in Spark SQL, and the particular problems associated with making time window-based aggregations work as expected. As a fairly advanced session we will assume familiarity with Spark and will not go over any aspects of its architecture.
The Speaker(s)