Quantcast
Channel: planet.communia
Viewing all articles
Browse latest Browse all 34

Recipe to migrate Drupal 7 Organic Groups to Drupal 9 Group

$
0
0
Recipe to migrate Drupal 7 Organic Groups to Drupal 9 GroupkintaSun, 10/03/2021 - 22:43

Despite there are some guides about migrating from OG to Group, as the well written https://thinktandem.io/blog/2018/03/30/migrating-drupal-7-organic-groups-to-drupal-8-group/ , and other guidelines such as https://www.drupal.org/project/group/issues/2826544 companion of https://gitlab.com/geeks4change/hubs4change/h4c/-/tree/608842e73cc02c45deede42556693a9db2a7fc37/modules_dev/h4c_migrate/config/install or https://www.drupal.org/node/2797845 . It's really adapted to your environment how you gonna carry on.

This is a short and crash guide that we at communia have done to complete the migrations. When Carrying on this migration must be estimated if the effort of writing code, in a form of migrate plugin classes or yml's, is worth enough than cooking csv's using spreadsheet skills. We've done both.

After importing everything from a standard drupal migration from 7 to 8(with what core offers), the groups types must be created, also their group roles and their group content types in a manual way. The site to be recreated is a news portal that takes headlines from several sites, and each blog or external blog is administrated in group, so being a group can be useful to add dynamic members or authors. So in the example I need to create two group types ( for each node enabled as organic group in d7), bundles are (machine)named:

  • blog(it was wrongly named previously bloc). I want also to add a role for each group type as admin to add granularity in permissions. So is needed to create the role admin for each type named:
  • blog_group-admin for blog group types.

I also marked the automatic role assignment for creators as admin role. It carries on a little of complexity in membership migrations as we must take care that memberships aren't duplicated (hence the plugin created below), as one membership will be created when groups are created on groups migration step, and another one when we import the memberships on memberships migration step.

So we need a simple module to add the migration configurations in yml, and it will aid us to create some extra plugins and hang on some hooks as hook_migrate_prepare_row. The module tree:

.├── group-content-export.csv.template├── group-memberships.csv.template├── migrations│   ├── d7_node_complete_og_to_canal_blog_extern.yml│   ├── d7_node_complete_og_to_canal_proposta_externa.yml│   ├── rename_d7_node_complete_blog_post.yml│   ├── rename_d7_node_complete_podcast.yml│   ├── upgrade_d7_node_og_blog_content.yml│   ├── upgrade_d7_node_og_blog_translation.yml│   ├── upgrade_d7_node_og_blog.yml│   └── upgrade_d7_user_og_blog_memberships.yml├── og_migrate_group.info.yml├── og_migrate_group.module├── README.md└── src└── Plugin└── migrate├── process   ├── GroupCreator.php   ├── SkipCreator.php   └── SkipOnNotEmpty.php

Go to gitlab.com to get the actual code for each file: https://gitlab.com/communia/og_migrate_group

So now let's yamling.

Groups

As Organic groups usually was assigned to nodes (if that's not the case cook it as you need), you 'll see it will be based in d7_node migration, so d7_node must be migrated before.

migrations/upgrade_d7_node_og_blog.yml:

langcode: enstatus: truedependencies: {  }id: upgrade_d7_node_og_blogclass: Drupal\migrate\Plugin\Migrationfield_plugin_method: nullcck_plugin_method: nullmigration_tags:  - 'Drupal 7'migration_group: insert_grouplabel: 'Insert OG Bloc group'source:  plugin: d7_node  # The node bundle that must be get from source  node_type: blocprocess:  type:    plugin: default_value    # put blog as group type (it can be simple mapped here but this is to show how default values work)    default_value: blog  id: nid  # The next uid will be the creator, if default role is marked it will be assigned to this user in group realm  uid: node_uid  label: title  path: alias  field_description: body  # An illustration of how an image in a group could be used, images must be migrated before as usual (in this example it is not using media module).  field_imatge_blogextern:    plugin: sub_process    source: field_imatge_blogextern    process:      target_id: fid      alt: alt      title: title      width: width      height: height  # Here an illustration of how a term could be assigned, once again terms must exist before migrating this.  field_tipus_bloc:    plugin: sub_process    source: taxonomy_vocabulary_7    process:      target_id:        plugin: entity_lookup        source: tid        value_key: tid        bundle_key: vid        bundle: vocabulary_7        entity_type: taxonomy_term  revision_uid: revision_uid  revision_log: log  revision_timestamp: timestampdestination:  # It's a group  plugin: 'entity:group'migration_dependencies: null

Once module is installed this migration can be imported with:

vendor/drush/drush/drush mim upgrade_d7_node_og_blog

and rolled back with:

vendor/drush/drush/drush migrate:rollback upgrade_d7_node_og_blog

if error locks any migration state saying it's currently importing (when is not) it can be fixed with:

drush migrate:reset-status upgrade_d7_node_og_blog

It's common that sometimes a crufty translation exists, in this example we found two extra fields that covered the translation, instead of using entity_translate or node_translate so field_titolbloces means title field in spanish and field_descripciobloces means field_description in spanish . This configuration will update existing groups translating it. Must be noted that group type must be marked as translatable editing the group type settings. migrations/upgrade_d7_node_og_blog_translation.yml:

langcode: enstatus: truedependencies: {  }id: upgrade_d7_node_og_blog_translationclass: Drupal\migrate\Plugin\Migrationmigration_tags:  - 'Drupal 7'migration_group: insert_grouplabel: 'Insert OG blog group translation'migration_dependencies:  required:    - upgrade_d7_node_og_blogsource:  plugin: d7_node  node_type: 'bloc'process:  # Will update the previously imported groups  id:    plugin: migration_lookup    source: nid    migration: upgrade_d7_node_og_blog  # Where can be found the label in spanish  label:    plugin: get    source: field_titolbloces  # Where can be found the description in spanish  field_description:    plugin: get    source: field_descripciobloces  # default value of langcode  langcode:    plugin: default_value    default_value: es  type:    plugin: default_value    default_value: blogdestination:  plugin: 'entity:group'  translations: true

This migration can be imported with:

vendor/drush/drush/drush mim upgrade_d7_node_og_blog_translation

and rolled back with:

vendor/drush/drush/drush migrate:rollback upgrade_d7_node_og_blog_translation

if error locks any migration state saying it's currently importing (when is not) it can be fixed with:

drush migrate:reset-status upgrade_d7_node_og_blog_translation

Memberships

As Organic groups memberships usually was assigned to users (if that's not the case cook it as you need), you 'll see it will be based in d7_user migration, so d7_user must be migrated before.

I found that here is easier to use csv, than to carry on too much code to take from here and there, so I have created a view of og memberships with data export in source site. View fields are:

  • OG membership: Og membership ID (id)
  • OG membership: Entity id (uid)
  • OG membership: Group ID (gid)
  • (Content entity referenced from og_user_node) Content: Identificator (UID) of the author (gid_uid)

View relationships are:

  • OG user node target id. (Content entity referenced from og_user_node) : A bridge to the Content entity that is referenced via og_user_node
  • OG membership: User from OG membership (REQUIRED)

If you prefer the sql of that view was:

SELECT og_membership.id AS id, og_membership.etid AS og_membership_etid, og_membership.gid AS og_membership_gid, node_og_membership.uid AS node_og_membership_uid, 'membership_export:page' AS view_nameFROM{og_membership} og_membershipLEFT JOIN {node} node_og_membership ON og_membership.gid = node_og_membership.nidLEFT JOIN {field_data_og_user_node} node_og_membership__field_data_og_user_node ON node_og_membership.nid = node_og_membership__field_data_og_user_node.og_user_node_target_id AND (node_og_membership__field_data_og_user_node.entity_type = 'user' AND node_og_membership__field_data_og_user_node.deleted = '0')LEFT JOIN {users} og_user_node_node ON node_og_membership__field_data_og_user_node.entity_id = og_user_node_node.uidLEFT JOIN {field_data_og_group_ref} node_og_membership__field_data_og_group_ref ON node_og_membership.nid = node_og_membership__field_data_og_group_ref.og_group_ref_target_id AND (node_og_membership__field_data_og_group_ref.entity_type = 'user' AND node_og_membership__field_data_og_group_ref.deleted = '0')LEFT JOIN {users} og_group_ref_node ON node_og_membership__field_data_og_group_ref.entity_id = og_group_ref_node.uidINNER JOIN {users} users_og_membership ON og_membership.etid = users_og_membership.uid AND (og_membership.entity_type = 'user')LEFT JOIN {profile} profile_users ON users_og_membership.uid = profile_users.uid

Anyway it must output what's in group-memberships.csv.template. Here the csv fields

  • id The id of the membership(rarely used, but preserved)
  • uid, The user id of the membership (so the member).
  • gid, The id of the group that user is member.
  • gid_uid, The creator of the group (not used in migration as we take it with a dedicated migration plugin).
"id","uid","gid","gid_uid""46","1","14829","1""50","4","14814","4""52","12","14834","1"

And here the migration that carries on:

langcode: enstatus: truedependencies: {  }id: upgrade_d7_user_og_blog_membershipsclass: Drupal\migrate\Plugin\Migrationfield_plugin_method: nullcck_plugin_method: nullmigration_tags:  - 'Drupal 7 og'migration_group: insert_grouplabel: 'Insert Group Membership'source:  plugin: csv  # Must be obtained from source with some exporter (e.g. views data export) and placed in this path  path: 'public://imports/group-export.csv'  header_row_count: 1  ids:    -      id  column_names:    -      id: Identifier    -      uid: UserID    -      gid: Group  delimiter: ','  enclosure: '"'process:  label: uid  group_creator:  # TO illustrate a plugin (not useful in the process),  that transforms from a group id to the uid of the group author.    -      plugin: group_creator      source: gid#    -#      plugin: callback#      callable: var_dump  # Now will check that group is there via previous migration lookup (the first one we did):  gid:    -      plugin: migration_lookup      migration: upgrade_d7_node_og_blog      no_stub: true      source: gid    -      plugin: skip_on_empty      method: row      message: 'gid is missing'  entity_id:    # To illustrate how the previous `group_creator` could be injected here( not useful):    #-    #  plugin: callback    #  callable: var_dump    #  source: "@group_creator"    # The plugin created to stop creating membership if current row user uid is the same as the group creator, to prevent duplicated members:    -      plugin: skip_creator      user_key: uid      gid_key: gid      method: row    # Finally check if user uid exists (probably if we are in this process point):    -      plugin: migration_lookup      migration: upgrade_d7_user      no_stub: true      source: uid    -      plugin: skip_on_empty      method: row      message: 'uid is missing'  # We hard code the role for all the migrated members:  group_roles:    plugin: default_value    default_value: blog-group_admin  type:    plugin: default_value    default_value: blog-group_membershipdestination:  # Memberships are considerated group_content also.  plugin: 'entity:group_content'migration_dependencies:  optional:    - upgrade_d7_node_og_blog    - d7_user

Note the use of skip_creator plugin to stop creating membership if current row user uid is the same as the group creator, to prevent duplicated members. The code of src/Plugin/migrate/process/SkipCreator.php is:

namespace Drupal\og_migrate_group\Plugin\migrate\process;use Drupal\migrate\MigrateExecutableInterface;use Drupal\migrate\ProcessPluginBase;use Drupal\migrate\MigrateException;use Drupal\migrate\MigrateSkipRowException;use Drupal\migrate\MigrateSkipProcessException;use Drupal\migrate\Row;/** * Skip if user_key as uid is the creator of group with gid as in gid_key. * * @MigrateProcessPlugin( *   id = "skip_creator" * ) * * To skip if creator use the following: * * @code * field_creator: *   plugin: skip_creator *   user_key: uid *   gid_key: gid *   method: row * @endcode * */class SkipCreator extends ProcessPluginBase {  /**   * Stops processing the current property when user_key as uid is the creator of group with gid as in gid_key.   *   * @param mixed $value   *   The input value.   * @param \Drupal\migrate\MigrateExecutableInterface $migrate_executable   *   The migration in which this process is being executed.   * @param \Drupal\migrate\Row $row   *   The row from the source to process.   * @param string $destination_property   *   The destination property currently worked on. This is only used together   *   with the $row above.   *   * @return mixed   *   The input value, $value, if it is not empty.   *   * @throws \Drupal\migrate\MigrateSkipProcessException   *   Thrown if the source property is not set and rest of the process should   *   be skipped.   */  public function process($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {    $group = \Drupal::entityTypeManager()      ->getStorage('group')      ->load($row->getSource()[$this->configuration["gid_key"]]);    if (!$group) {      throw new MigrateException('gid not found.');    }    if ($group->getOwner()->id() == $row->getSource()[$this->configuration["user_key"]]) {      throw new MigrateSkipRowException("Creator is the same uid, so it exists");    }    return $value;  }  /**   * Skips the current row when user_key as uid is the creator of group with gid as in gid_key.   *   * @param mixed $value   *   The input value.   * @param \Drupal\migrate\MigrateExecutableInterface $migrate_executable   *   The migration in which this process is being executed.   * @param \Drupal\migrate\Row $row   *   The row from the source to process.   * @param string $destination_property   *   The destination property currently worked on. This is only used together   *   with the $row above.   *   * @return mixed   *   The input value, $value, if it is not empty.   *   * @throws \Drupal\migrate\MigrateSkipRowException   *   Thrown if the source property is not set and the row should be skipped,   *   records with STATUS_IGNORED status in the map.   */  public function row($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {    $group = \Drupal::entityTypeManager()      ->getStorage('group')      ->load($row->getSource()[$this->configuration["gid_key"]]);    if (!$group) {      throw new MigrateException('gid not found.');    }    if ($group->getOwner()->id() == $row->getSource()[$this->configuration["user_key"]]) {      throw new MigrateSkipRowException("Creator is the same uid, so it exists");    }    return $value;  }}

So now we have groups and their members. Now we can start thinking about the content.

Content

Here instead of doing everything at once (create nodes and add as content to groups) as in https://thinktandem.io/blog/2018/03/30/migrating-drupal-7-organic-groups-to-drupal-8-group/ , it will be based in a previously migrated node set, as it was did in previous step where from users it created group memberships.

Let's create the group content first setting that content as available in group type settings/content, in this example blog group available content will be proposta, blog_post(as blog entry), podcast_episode(as audio entry). Despite we could create a new plugin which transforms from nid to title or uid, may be easier to do it in source view group_content csv, so including these properties. Relevant view used to obtain source data will be:

Relations:

  • OG membership: Content from OG membership: 'og_membership_related_node'; REQUIRED
  • Entity Reference : Referenced Entity: 'og_group_ref_target_id';

Fields:

  • id, label: 'og_membership_id'
  • (relationship: og_membership_related_node_1)->nid , label:content_nid
  • (relationship: og_membership_related_node_1)->type, label: content_bundle
  • (relationship: og_membership_related_node_1)->uid, label: content_uid
  • (relationship: og_membership_related_node_1)->title, label: content_title
  • gid, label: 'gid'
  • (relationship:og_group_ref_target_id)->type, label: group_type

(With the filters and sorts adapted to your case, to prevent listing non-migratable content)

This will output

"og_membership_id","content_nid","content_bundle","content_uid","content_title","gid","group_type""13397","59546","blog","438","Good news","22477","bloc""13426","77","podcast","6","El sistema universitari","22447","bloc""13427","86","proposta","16","Estrobaca","22455","bloc""13428","87","blog","8","ssl","22449","bloc"

Here the yml that will process this csv:

langcode: enstatus: truedependencies: {  }id: upgrade_d7_node_og_blog_contentclass: nullfield_plugin_method: nullcck_plugin_method: nullmigration_tags:  - CSVmigration_group: insert_grouplabel: 'blog Group Content Migration from CSV'source:  plugin: csv  # Must be obtained from source with some exporter (e.g. views data export) and placed in this path  path: 'public://imports/group-content-export.csv'  header_row_count: 1  ids:    -      og_membership_id  column_names:    -      og_membership_id: Identifier    -      content_nid: NodeID    -      content_bundle: bundle    -      content_uid: uid    -      content_title: title    -      gid: group    -      group_type: GroupType  delimiter: ','  enclosure: '"'process:  label:    # TO show the current item to debug    #-    #  plugin: callback    #  callable: var_dump    #  source: content_title    -      plugin: get      source: content_title  type:    plugin: static_map    source: content_bundle    bypass: false    # Customize this to match your values and group content plugins    map:      proposta: blog-group_node-proposta      blog: blog-group_node-blog_post      podcast: blog-group_node-podcast_episode    default_value: blog-group_node-blog_post  gid: gid  entity_id:    # To skip if element is not found (may be merged during translation    # nodes migration fixes)    -      plugin: get      source: content_nid    -      plugin: entity_lookup      entity_type: node      value_key: nid    -      plugin: skip_on_empty      method: row  uid: content_uiddestination:  plugin: 'entity:group_content'migration_dependencies: null

And that's all, after that Groups, Memberships and Content will be migrated.

Bonus track - rename machine name of a bundle when migration has been already executed.

First of all we create the bundle giving the desired name, and with the module field_tools we can make the clone of view_modes first and after that the fields clones. Once it is identical, we must rollback the migration of the content a given bundle: drush migrate:rollback d7_node_complete:podcast and then copy from web/sites/default/files/config__secret_directory_that _must_be_uniquely_and_obscured_from_public_found_at_settings/sync/migrate_plus.migration.upgrade_d7_node_complete_podcast.yml to a dedicated custom module in its migrations directory, then modify file setting the destination to desired bundle machine_name, also must be changed the id and filename according to what you are doing(in this case we set to rename_d7_node_complete_podcast), then run drush migrate:rollback d7_node_complete:podcast --verbose && drush mim rename_d7_node_complete_podcast --verbose to import again the content that we want. Any configuration over it must be executed once again, for example url_redirects migrations.

 

 

Bonus track 2 - node feeds in D7 to feeds_feed entity in D8

First create the feed types and add the fields as it was defined in source nodes that were acting as importers, then edit add the migration configuration, two working samples are in migrations:

  • d7_node_complete_og_to_canal_blog_extern.yml
  • d7_node_complete_og_to_canal_proposta_externa.yml

Both are based on the stock migration configurations that migrateplus has created and exported in config sync directory: upgrade_d7_node_complete_[bundle_of_source] . You can edit to create the feed entity instead of a node. Be careful to rollback previous imported bundle nodes migrations that match (in example it was feed and proposta_feed bundles). Flush plugins cache, and after that drush mim d7_node_complete_node_to_[bundle_of_source] will do the magic. This is the first step, Now you have two options because  feed source url is not easily picked.

One way and recommended as it can be automatizated, is using a new source plugin to fix the feed urls, look at commit  this is a new plugin that will lookup the feed in source d7 feed table and "magically" will provide the url using some config as this:

 

langcode: en
status: true
dependencies: {  }
id: d7_feed_sources
class: Drupal\migrate\Plugin\Migration
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7 feeds'
  - 'resync'
migration_group: resync
label: 'Drupal 7 Feeds sources'
source:
  plugin: d7_feed
process:
  fid:
    -
      plugin: entity_lookup
      source: title
      value_key: title
      entity_type: feeds_feed
#    -
#      plugin: callback
#      callable: var_dump
  source: source
destination:
  plugin: 'entity:feeds_feed'
migration_dependencies:
  optional:
    - d7_node_complete_og_to_canal_blog_extern
    - d7_node_complete_og_to_canal_proposta_externa

 

The other way is using csv in a manual way,  first instead of struggling too much I end up creating a view in source that outputs to csv these fields:

  • Content: title (title)
  • Feeds source: Source (source)
  • Content: type (bundle)
  • Content: Nid (nid)

Which csv I copied to public://imports/export-feeds.csv And created the auxiliary migration configuration to import from csv:

langcode: enstatus: truedependencies: {  }id: fix_feeds_sourcesclass: Drupal\migrate\Plugin\Migrationfield_plugin_method: nullcck_plugin_method: nullmigration_tags:  - 'Drupal 7 feeds'migration_group: fix_feedslabel: 'Fix Feeds sources'source:  plugin: csv  path: 'public://imports/export-feeds.csv'  header_row_count: 1  ids:    -      nid  column_names:    -      title: title    -      source: Source    -      bundle: bundle    -      nid: Identifier  delimiter: ','  enclosure: '"'process:  fid:    -      plugin: entity_lookup      source: title      value_key: title      entity_type: feeds_feed#    -#      plugin: callback#      callable: var_dump  source: sourcedestination:  plugin: 'entity:feeds_feed'migration_dependencies:  optional:    - d7_node_complete_og_to_canal_blog_extern    - d7_node_complete_og_to_canal_proposta_externa

then drush mim fix_feeds_sources again. Then create the feed mappings as it was done in source(or modify as you want). And start importing


Viewing all articles
Browse latest Browse all 34

Trending Articles