COLE: A Hierarchical Generation Framework for
Multi-Layered and Editable Graphic Design

Peidong Jia1,2      Chenxuan Li1,2      Yuhui Yuan1,3      Zeyu Liu2      Yichao Shen2      Bohan Chen2     
Xingru Chen2    Yinglin Zheng2      Dong Chen    Ji Li    Xiaodong Xie    Shanghang Zhang    Baining Guo   
1 joint core contribution    2 interns at microsoft    3 project lead   
Microsoft Research Asia         Peking University
Full-res Paper(140M) arXiv BenchSVGs BenchPNGs Data Video
Image 1
Design a recruiting post for a graphic designer job opening. Encourage interested candidates to send their CV to yourname@email.com
Image 1
Design a gift certificate for a travel insurance company promoting their services with a 30% discount offer. The certificate should convey a sense of safety and protection while travelling.
Image 1
Create a series of posters for a fictional Ancient Greek Olympics, with each poster highlighting a different sport like chariot racing or discus throw, styled in traditional Greek pottery art.
Image 1
Design an attractive poster for a Mars botanical garden tour. Illustrate a greenhouse with a variety of extraterrestrial plants. Use vibrant colors to depict the potential biodiversity on Mars.
Image 1
Create an enticing poster for a classical orchestra concert in the heart of the Amazon Rainforest. (...). Incorporate subtle hints of wildlife and flora within the musical instruments.
Image 1
Design a promotional poster for a Titanic-themed immersive theatre experience. (...). It should also tease some of the scenes from the play, such as the grand dining hall and the ship's deck.
Image 1
Create a coupon that promotes a garden design service offered by Yard Culture. The coupon is a gift voucher valued at $49 and can only be used once. It is valid until November 15th. The coupon number is No:7496-4587.
Image 1
Promote a stand-up comedy show happening on November 12th, 20xx.
Interpolate start reference image.
Design an animated digital poster for a fantasy concert where Joe Hisaishi performs on a grand piano on a moonlit beach. The scene should be magical, with bioluminescent waves and a starry sky. The poster should invite viewers to an exclusive virtual event.
Interpolate start reference image.
Create an Instagram story promoting a special Christmas offer for a chocolate drink. The offer includes a 20% discount and encourages customers to order now.
Interpolate start reference image.
Create a customer testimonial advertisement showcasing a pink clothing collection. The advertisement includes a positive review from Jenny Wilson.
Interpolate start reference image.
Design an alluring poster for a blues concert in a clearing of the Amazon Rainforest. (...). Use a moody, midnight blue palette and incorporate silhouettes of blues instruments and nocturnal wildlife.
Interpolate start reference image.
Design an advertisement card to promote the sale of premium quality school equipment, with a focus on enhancing the learning experience for students.
Interpolate start reference image.
Create a business card for a travel agency that offers vacation planning services. The design features a red suitcase to represent travel. The contact information includes phone number +123-456-7890 and website www.yourwebsite.com.
Interpolate start reference image.
Create a poster advertising a corner spice shop with a wide range of spices and ingredients.
Interpolate start reference image.
Create a captivating poster for an upcoming documentary about the mysteries of the Egyptian pyramids. (...). Use a muted color palette to imitate the sands of Egypt and the age-old structures.
Image 1
Create an invitation card for an origami class that teaches the attendees how to make flowers and animals. The class is from 5 PM to 6.30 PM on Jan 9th. More information is available on the website myorigami.net.
Image 1
Design a festive Christmas greeting card with a cute and playful illustration that features a winter scene with trees, snowflakes and a festive greeting. The card is part of the HelloChristmas series and features a red and white color scheme.
Image 1
Design an elegant invitation for a wedding dinner rehearsal, with a romantic and festive feel. The event date is August 10, 2022 and the names are Dana and Alan.
Image 1
Create a Valentine's Day greeting card with a romantic and playful design that expresses love. The text 'Happy Valentine's Day' and 'I Love You' are included.
Image 1
Design a thank you card with a floral theme and a message expressing gratitude. The card is intended to be used for business purposes and features the company logo.
Image 1
Design a premium and high quality emblem for a coffee house that emphasizes on the freshness and aroma of the coffee served. The text 'Coffee & More' is included to show that there is more to offer than just coffee.
Image 1
Design a poster to promote a charity bike ride event to raise funds and support for a humanitarian cause. The event will be held on June 26th and the website for registration and donation is www.funbike.com.
Image 1
Promote camping equipment for adventurous couples who enjoy spending time in nature and camping. Encourage them to order now through the ad on Instagram.

Abstract

Graphic design, which has been evolving since the 15th century, plays a crucial role in advertising. The creation of high-quality designs demands design-oriented planning, reasoning, and layer-wise generation. This intricate task involves understanding vague intentions and faithfully generating multi-layered visual elements, including the background, decorations, objects, and fonts, among others. It also requires layout planning for all elements and visual reasoning to generate all visual elements that satisfy visual design principles.

Unlike the recent CanvaGPT, which integrates GPT-4 with existing design templates to build a custom GPT, this paper introduces the system—a hierarchical generation framework designed to comprehensively address these challenges. This system can transform a vague intention prompt into a high-quality multi-layered graphic design, while also supporting flexible editing based on user input. Examples of such input might include directives like "design a poster for Hisaishi's concert."

The key insight is to dissect the complex task of text-to-design generation into a hierarchy of simpler sub-tasks, each addressed by specialized models working collaboratively. The results from these models are then consolidated to produce a cohesive final output. Our hierarchical task decomposition can streamline the complex process and significantly enhance generation reliability.

Our system comprises multiple fine-tuned Large Language Models (LLMs), Large Multimodal Models (LMMs), and Diffusion Models (DMs), each specifically tailored for design-aware layer-wise captioning, layout planning, reasoning, and the task of generating images and text. Furthermore, we construct the benchmark to demonstrate the superiority of our system over existing methods in generating high-quality graphic designs from user intent. Last, we present a Canva-like multi-layered image editing tool to support flexible editing of the generated multi-layered graphic design images.

We perceive our system as an important step towards addressing more complex and multi-layered graphic design generation tasks in the future.



Method Overview


Design Diversity with Different Random Seeds


Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.



Flexible Editing Results


Interpolate start reference image.

Original

Interpolate start reference image.

Edit text only

Interpolate start reference image.

Edit object only

Interpolate start reference image.

Finally result

Interpolate start reference image.

Original

Interpolate start reference image.

Edit text only

Interpolate start reference image.

Edit object only

Interpolate start reference image.

Finally result

Edit Demo Video



More Results


Interpolate start reference image.
Create a gift card for a store's Boxing Day sale. The gift card is worth $200 and can be used for shopping on Boxing Day.
Interpolate start reference image.
Create an Instagram post that encourages people to vote and emphasizes the importance of their vote. Use a prominent illustration of a man casting his ballot, and include the text \"Your vote matters more than ever! Your Voice, Your Future. Vote for us.
Interpolate start reference image.
Construct an eye-catching poster for an upcoming academic conference on Quantum Physics. The design should encapsulate the mysterious and profound nature of the subject. (...). Make sure the conference details and keynote speakers are clearly visible.

Interpolate start reference image.
Create a Youtube thumbnail for promoting an antique store subscription service. Showcase the variety of unique vintage goods available in the store.

Your image alt
Design a Facebook event cover for a winter decoration workshop featuring lectures and demonstrations by Miranda Yates, Sam Knightly, Zachary Evans, and Veronica Floyd on December 24 at Lafayette Studio.

Interpolate start reference image.
Create a Facebook cover image to announce a flower exhibition event.

Interpolate start reference image.
Create a poster for a chess tournament set in an ancient Roman arena. Immerse the audience in a world where strategic battles are fought not with swords, but with chess pieces. Evoke the grandeur of the Colosseum with a backdrop of sandstone hues and imposing Roman architecture.

Interpolate start reference image.
Create a Valentine’s Day greeting card with a cute and romantic design. The card includes the phrase ’Happy Valentine’s Day’ and ’Much love to you!’

Interpolate start reference image.
Design a holiday-themed gift card to promote the sale of gift boxes and packaging for the Christmas in July event. Include the website address www.yourbrand.com and store name.

Your image alt
Create a birthday card with a festive and colorful design featuring a cake with fruits. The card is an invitation to a birthday party happening on June 22nd from 8pm to 11pm at 123 st. Any city 4567.

Interpolate start reference image.
Create a business card for a car service company that highlights their commitment to providing reliable and efficient car maintenance and repair services. The slogan ’Give your car wings’ emphasizes the quality of service provided.

Interpolate start reference image.
Create an Instagram ad for a toy store’s Black Friday and Cyber Monday sale with a focus on the discount of 45% off on selected toys.

Interpolate start reference image.
Create a Women's Day greeting card with signs of female gender, celebrating women and their achievements. The design should be beautiful and include floral elements. The card should be suitable for store use and have a message of 'Happy Women's Day'.

Interpolate start reference image.
Design an elegant invitation for a wedding dinner rehearsal, with a romantic and festive feel. The event date is August 10, 2022 and the names are Dana and Alan.

Your image alt
Design a gift certificate for a travel insurance company that offers a 30% discount for all kinds of travel.

Interpolate start reference image.
Create a Thanksgiving gift certificate with an elegant and sophisticated design that offers a 50% discount on decorations. The message wishes a happy Thanksgiving to the recipient.

Interpolate start reference image.
Design a colorful store logo with an illustration of leaves to represent a fresh and vibrant shopping experience. The store may be named TB.

Interpolate start reference image.
Create a LinkedIn post with a cartoon illustration to promote an article about the benefits and growing popularity of remote work from home. The post links to a website for more information about the topic and is associated with a specific company and CEO John Doe.,

Interpolate start reference image.
Design a poster to promote a charity bike ride event to raise funds and support for a humanitarian cause. The event will be held on June 26th and the website for registration and donation is www.funbike.com.

Your image alt
Create an advertisement poster for a college merchandise store that has been operating since 1991. The design should focus on college and university branded gear, including clothes, backpacks, and bags.

Interpolate start reference image.
Create an Instagram story to advertise an antique vase available at a store with a brown color scheme. The promotion offers up to a 50% discount and is only available from October 22-25, 2023.

Interpolate start reference image.
Create an Instagram story to promote a special offer of 30% discount on the autumn collection. Encourage users to visit the website by including a link in bio.

Interpolate start reference image.
Create an Instagram story frame with a futuristic and fantasy-like design that features a man in an astronaut helmet and a space-themed background. The text #ThisIsMetaLife and SPACEDOUT are included.

Interpolate start reference image.
Create a recruiting Instagram post to advertise a job vacancy for a UI/UX designer. Encourage passionate individuals to apply and provide a link to apply in the bio.

Your image alt
Promote a personal styling service for summer vacation outfits on Instagram. The contact number is +00 1357 2468 and website is www.website.com.

Interpolate start reference image.
Create a poster for a Mars charity marathon. Highlight the challenging environment of Mars while also symbolizing the spirit of resilience and determination. Encourage viewers to donate for the cause of making life sustainable on Mars.

Interpolate start reference image.
Create an Instagram story ad for an antique shop, promoting the sale of a timeless typewriter. Encourage customers to visit the store by swiping up on the story.

Interpolate start reference image.
Design a Christmas sale poster with a fun and festive look. The poster should advertise up to 50% off on Christmas accessories, with a limited time offer.

Interpolate start reference image.
Design a dramatic poster for an opera performance in a natural Amazon Rainforest amphitheater. (...). Show a dramatic scene unfolding on the forest stage.

Your image alt
Design a Facebook cover image that promotes the importance of conservation and the preservation of nature. The text 'Save Our Oxygen Source' emphasizes the significance of forests in producing oxygen.



BibTeX


      @misc{jia2024cole,
        title={COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design}, 
        author={Peidong Jia and Chenxuan Li and Yuhui Yuan and Zeyu Liu and Yichao Shen and Bohan Chen and Xingru Chen and Yinglin Zheng and Dong Chen and Ji Li and Xiaodong Xie and Shanghang Zhang and Baining Guo},
        year={2024},
        eprint={2311.16974},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
  }

Acknowledgements


Website adapted from the following template.