Strange action errors in mirrored projects?! #32232

somera · 2024-10-10T11:15:28Z

Description

After update to 1.22.3 I see many of errors

in many mirrored projects?!

And than I saw this

or

What was activated with 1.22.3 release? What happens here?

Gitea Version

1.22.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.47.0

Operating System

Ubuntu 22.04.3 LTS x86_64

How are you running Gitea?

gitea-1.22.3-linux-amd64

Database

PostgreSQL

somera · 2024-10-10T11:18:57Z

I see that

are activated. But not for all mirror.

And I didn't activate it.

I have currently 319 mirrors with

select
	*
from
	repository
where
	is_mirror = true
and
	(num_action_runs > 0 or num_closed_action_runs > 0);

How can I check in the DB which mirrors has activated actions? I don't know the structure and I didn't find other corresponding columns.

somera · 2024-10-10T11:25:32Z

How can I fix it?

I can't go over the whole list and do it manualy.

When I deacticate the Actions in one specific project than I see the error

and when I click on the x I get 404 error.

somera · 2024-10-10T17:22:59Z

@lunny ?

lunny · 2024-10-10T17:58:54Z

@lunny ?

I will take a look today

somera · 2024-10-12T10:43:24Z

@lunny I see more and more mirrors with ...

lunny · 2024-10-13T20:17:13Z

I have no idea, maybe it's related to #32120 . @yp05327 @Zettat123

Zettat123 · 2024-10-14T02:27:00Z

I see that

are activated. But not for all mirror.

And I didn't activate it.

Do you mean that the "Actions" unit of some repositories are automatically enabled after upgrading to v1.22.3? If so, I think this issue is not related to #32120. Did you upgrade from v1.22.2 to v1.22.3?

somera · 2024-10-14T08:50:19Z

I see that

are activated. But not for all mirror.
And I didn't activate it.

Do you mean that the "Actions" unit of some repositories are automatically enabled after upgrading to v1.22.3? If so, I think this issue is not related to #32120. Did you upgrade from v1.22.2 to v1.22.3?

Yes. After upgrade from 1.22.2 to 1.22.3 I see

And Actions are for the Mirrors activated. It makes no sense.

somera · 2024-10-14T11:24:41Z

Example

I didn't activate the Actions for all the affected projects.

And the red x is now after upgrade to 1.22.3.

When I add new mirror, Actions are activated -> is this right? Was this activated in 1.22.2 too?

Should the Actions be activated or not? If yes, what happens here for mirrors?
Why I see the whole failed old history?
Why will be the red x shown for mirrors? Looks like an error for me. What changed here from 1.22.2 to 1.22.3?
Is something broken is the main question for me.

somera · 2024-10-14T12:23:05Z

I compared 1.22.2 (older mirror status) and 1.22.3 instances for same mirrors.

1.22.2 -> actions tab is not activated, no red x visible
1.22.3 -> actions tab is activated, red x visible for "failed" actions in some projects

somera · 2024-10-14T12:49:05Z

Strange.

I imported an existing mirror in 1.22.2 in other orga on the same gite instance. And the new mirror has activated actions tab. In the old mirror the actions tab is not activated. When I update the old mirror the actions stay disabled.

What happens here?

Zettat123 · 2024-10-15T01:57:50Z

When I add new mirror, Actions are activated -> is this right? Was this activated in 1.22.2 too?

Yes. New repositories (including mirrors) created in versions after v1.22 will have "Actions" enabled by default. See #27482. However, I don't think repositories created before v1.22 will automatically enable "Actions" after upgrading. I'll do some testing to confirm.

I've done my testing and can now confirm that mirror repositories created before v1.22 will not have "Actions" enabled after Gitea upgraded to v1.22. So I think repositories with "Actions" enabled should be created after v1.22.0

There is a proposal to disable "Actions" for mirror repositories by default. I support this idea.

Disable Action unit by default when mirroring a repo? #30350

yp05327 · 2024-10-15T04:18:13Z

When does these jobs start being triggered?

I can see some of them, but I'm sorry I don't know the language in your screenshot.

Does this mean last month?

And when did you upgrade the instance?

If the jobs are triggered before the update, it make sense. And I think this means before the update, the action unit is already enabled. But you said, they are disabled before the update, and enabled after the update, which is strange. And I have no idea about it yet.
But if they are all triggered after the update, it means during the update, action units was forcibly enabled. Then it should be a bug or something was wrong from Gitea side.

Zettat123 · 2024-10-15T04:37:08Z

But if they are all triggered after the update, it means during the update, action units was forcibly enabled. Then it should be a bug or something was wrong from Gitea side.

This is where I suspect a bug exists. But I can't find any evidence that Gitea forcibly enabled the "Actions" unit.

yp05327 · 2024-10-15T05:03:02Z

@Zettat123
Yes. I think it is the point, as there are several possible cases, so I asked, and then we can know what happened.

somera · 2024-10-15T06:22:11Z

When does these jobs start being triggered?

I can see some of them, but I'm sorry I don't know the language in your screenshot. Does this mean last month?

It's German. And it mean last month.

I'm wondering too, that I see this actions.

And when did you upgrade the instance?

~2 days aftrer 1.22.3 was released.

If the jobs are triggered before the update, it make sense. And I think this means before the update, the action unit is already enabled. But you said, they are disabled before the update, and enabled after the update, which is strange. And I have no idea about it yet. But if they are all triggered after the update, it means during the update, action units was forcibly enabled. Then it should be a bug or something was wrong from Gitea side.

But i didn't enable any actions. I checked a lot of mirrors and most of them has no activated actions.

somera · 2024-10-15T06:25:53Z

This https://github.com/auth0/go-auth0 is an mirror.

2 months ago

And 1.22.3 was upgraded on 09.10.2024

And here othe mirror with activated action but wihout any errors

Other mirror ...

I don't know how many are affected.

yp05327 · 2024-10-15T06:34:44Z

And here othe mirror with activated action but wihout any errors

This is because the trigger of these 2 workflows are creating PR to main branch and workflow_dispatch (manually excute)

ps: for @Zettat123, the workflow files:
https://github.com/stanford-oval/storm/tree/main/.github/workflows

yp05327 · 2024-10-15T06:37:15Z

So IMO, actually, there's no bug and the only thing you need is #30350

Zettat123 · 2024-10-15T06:43:22Z

And here othe mirror with activated action but wihout any errors

This is because the trigger of these 2 workflows are creating PR to main branch and workflow_dispatch (manually excute)

ps: for @Zettat123, the workflow files: https://github.com/stanford-oval/storm/tree/main/.github/workflows

I see. Actually, the "Actions" unit was enabled for all repositories, but only some of them had actions runs due to workflow triggers.

somera · 2024-10-15T06:45:45Z

Will the patch make it like it was before or does something still need to be fixed in the database?

yp05327 · 2024-10-15T06:49:05Z

In another way, it is not difficult to implement a new options when migrating the repo.
But how to handle these existing mirrors, adding a migration or a maintenance operation or doctor command?

Zettat123 · 2024-10-15T07:07:57Z

To batch disable the "Actions" unit of existing mirrors, users can use the API PATCH
/repos/{owner}/{repo} to update repos.

somera · 2024-10-15T07:59:01Z

In another way, it is not difficult to implement a new options when migrating the repo. But how to handle these existing mirrors, adding a migration or a maintenance operation or doctor command?

Doctor commands sounds good. I have ~20000 mirrors.

How can I check in the DB how many mirrors are at the moment affeced?

And when I disable Actions for some mirrors in the UI the red x and orange o stay.

somera · 2024-10-15T17:53:37Z

And today I see first system messages

I missing here info to the project.

2024/10/15 19:50:18 ...eb/routing/logger.go:102:func1() [I] router: completed GET /repo/search?sort=updated&order=desc&uid=1&team_id=undefined&q=&page=1&limit=15&mode=&archived=false for 192.168.178.21:0, 200 OK in 95.1ms @ repo/repo.go:556(repo.SearchRepo)
2024/10/15 19:50:22 ...ns/schedule_tasks.go:87:startTasks() [E] CreateScheduleTask: SetJob: yaml: line 15: did not find expected key
        /source/services/actions/schedule_tasks.go:87 (0x1ec5678)
        /source/services/actions/schedule_tasks.go:24 (0x25858bc)
        /source/services/cron/tasks_actions.go:63 (0x25858bd)
        /source/services/cron/tasks.go:111 (0x25824c7)
        /source/modules/graceful/manager.go:92 (0xaa9d26)
        /source/services/cron/tasks.go:88 (0x258223a)
        /source/services/cron/tasks.go:67 (0x2586d93)
        /usr/local/go/src/reflect/value.go:596 (0x4fccc5)
                Value.call: call(frametype, fn, stackArgs, uint32(frametype.Size()), uint32(abid.retOffset), uint32(frameSize), &regArgs)
        /usr/local/go/src/reflect/value.go:380 (0x4fbdd8)
                Value.Call: return v.call("Call", in)
        /go/pkg/mod/github.com/go-co-op/[email protected]/gocron.go:122 (0x2563904)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:87 (0x2561b31)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:213 (0x2562817)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:269 (0x25632b7)
        /usr/local/go/src/runtime/asm_amd64.s:1695 (0x47b0a0)
                goexit: BYTE    $0x90   // NOP

and a lot exceptions

2024/10/15 14:04:22 ...irror/mirror_pull.go:178:parseRemoteUpdateOutput() [W] parseRemoteUpdateOutput: unexpected update line " * [new ref]         refs/pull/798/head  -> refs/pull/798/head"
2024/10/15 14:04:22 ...irror/mirror_pull.go:178:parseRemoteUpdateOutput() [W] parseRemoteUpdateOutput: unexpected update line " * [new ref]         refs/pull/798/merge -> refs/pull/798/merge"
2024/10/15 14:05:22 ...ns/schedule_tasks.go:87:startTasks() [E] CreateScheduleTask: SetJob: yaml: line 15: did not find expected key
        /source/services/actions/schedule_tasks.go:87 (0x1ec5678)
        /source/services/actions/schedule_tasks.go:24 (0x25858bc)
        /source/services/cron/tasks_actions.go:63 (0x25858bd)
        /source/services/cron/tasks.go:111 (0x25824c7)
        /source/modules/graceful/manager.go:92 (0xaa9d26)
        /source/services/cron/tasks.go:88 (0x258223a)
        /source/services/cron/tasks.go:67 (0x2586d93)
        /usr/local/go/src/reflect/value.go:596 (0x4fccc5)
                Value.call: call(frametype, fn, stackArgs, uint32(frametype.Size()), uint32(abid.retOffset), uint32(frameSize), &regArgs)
        /usr/local/go/src/reflect/value.go:380 (0x4fbdd8)
                Value.Call: return v.call("Call", in)
        /go/pkg/mod/github.com/go-co-op/[email protected]/gocron.go:122 (0x2563904)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:87 (0x2561b31)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:213 (0x2562817)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:269 (0x25632b7)
        /usr/local/go/src/runtime/asm_amd64.s:1695 (0x47b0a0)
                goexit: BYTE    $0x90   // NOP


2024/10/15 14:06:22 ...ns/schedule_tasks.go:87:startTasks() [E] CreateScheduleTask: SetJob: yaml: line 15: did not find expected key
        /source/services/actions/schedule_tasks.go:87 (0x1ec5678)
        /source/services/actions/schedule_tasks.go:24 (0x25858bc)
        /source/services/cron/tasks_actions.go:63 (0x25858bd)
        /source/services/cron/tasks.go:111 (0x25824c7)
        /source/modules/graceful/manager.go:92 (0xaa9d26)
        /source/services/cron/tasks.go:88 (0x258223a)
        /source/services/cron/tasks.go:67 (0x2586d93)
        /usr/local/go/src/reflect/value.go:596 (0x4fccc5)
                Value.call: call(frametype, fn, stackArgs, uint32(frametype.Size()), uint32(abid.retOffset), uint32(frameSize), &regArgs)
        /usr/local/go/src/reflect/value.go:380 (0x4fbdd8)
                Value.Call: return v.call("Call", in)
        /go/pkg/mod/github.com/go-co-op/[email protected]/gocron.go:122 (0x2563904)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:87 (0x2561b31)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:213 (0x2562817)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:269 (0x25632b7)
        /usr/local/go/src/runtime/asm_amd64.s:1695 (0x47b0a0)
                goexit: BYTE    $0x90   // NOP


2024/10/15 14:07:22 ...ns/schedule_tasks.go:87:startTasks() [E] CreateScheduleTask: SetJob: yaml: line 15: did not find expected key
        /source/services/actions/schedule_tasks.go:87 (0x1ec5678)
        /source/services/actions/schedule_tasks.go:24 (0x25858bc)
        /source/services/cron/tasks_actions.go:63 (0x25858bd)
        /source/services/cron/tasks.go:111 (0x25824c7)
        /source/modules/graceful/manager.go:92 (0xaa9d26)
        /source/services/cron/tasks.go:88 (0x258223a)
        /source/services/cron/tasks.go:67 (0x2586d93)
        /usr/local/go/src/reflect/value.go:596 (0x4fccc5)
                Value.call: call(frametype, fn, stackArgs, uint32(frametype.Size()), uint32(abid.retOffset), uint32(frameSize), &regArgs)
        /usr/local/go/src/reflect/value.go:380 (0x4fbdd8)
                Value.Call: return v.call("Call", in)
        /go/pkg/mod/github.com/go-co-op/[email protected]/gocron.go:122 (0x2563904)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:87 (0x2561b31)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:213 (0x2562817)
        /go/pkg/mod/github.com/go-co-op/[email protected]/executor.go:269 (0x25632b7)
        /usr/local/go/src/runtime/asm_amd64.s:1695 (0x47b0a0)
                goexit: BYTE    $0x90   // NOP

somera · 2024-10-15T18:22:18Z

Same #30244

somera · 2024-10-15T18:38:51Z

Can I use this

; [actions]
;; Enable/Disable actions capabilities
;ENABLED = false

to disable action temporary?

And this

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Cleanup expired actions assets
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;[cron.cleanup_actions]
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;ENABLED = true
;RUN_AT_START = true
;SCHEDULE = @midnight

to cleanup all the action jobs?

yp05327 · 2024-10-16T23:57:14Z

It is a global setting for all repos, if you do not need to use actions in this instance, you can disable it globally.

Clean up only remove action job log, related data and artififacts, it will not delete those action jobs.

somera · 2024-10-17T05:17:05Z

It is a global setting for all repos, if you do not need to use actions in this instance, you can disable it globally.

Thx,

Clean up only remove action job log, related data and artififacts, it will not delete those action jobs.

hm.

Are you working on a solution to this problem?

The failed action jobs are flooding my system messages view.

yp05327 · 2024-10-17T06:32:22Z

For existing mirrors, no good solution from me.

As it is related to repo_unit table in DB.
If there's a record with unit type 10 (action), and it is not globally disabled, then it means action unit is enabled.
We can not simply remove these records in migration, as users may need it.
So we can only do it by ourselves (cli or sql) or add a new maintenance operations, but it is not a usually used feature I think, so maybe there's no worth to implement it.

↑ It is for the action jobs

And the commit status (x in dashboard repo list) is recorded in another table, and if you don't want to see them, maybe you also need to clean it.

So, as a summary, I have no good idea now for completely solving it for these existing mirrors.

somera · 2024-10-17T10:08:27Z

Understand. Means I have fix it for ~20.000 mirrors by myself. Or I do

To batch disable the "Actions" unit of existing mirrors, users can use the API PATCH /repos/{owner}/{repo} to update repos.

this with an script for all the mirrors?

I this case I should send only has_actions = false as an json? Right?

This will disable the action. And how can I than fix this

?

Zettat123 · 2024-10-18T02:01:58Z

I this case I should send only has_actions = false as an json? Right?

Yes. You only need to set the fields that need to be changed.

gitea/routers/api/v1/repo/repo.go

Lines 1018 to 1027 in 9116665

    
           if opts.HasActions != nil && !unit_model.TypeActions.UnitGlobalDisabled() { 
        
           	if *opts.HasActions { 
        
           		units = append(units, repo_model.RepoUnit{ 
        
           			RepoID: repo.ID, 
        
           			Type:   unit_model.TypeActions, 
        
           		}) 
        
           	} else { 
        
           		deleteUnitTypes = append(deleteUnitTypes, unit_model.TypeActions) 
        
           	} 
        
           }

somera · 2024-10-18T13:20:44Z

@Zettat123 I did this for ~3500 mirrors. Looks good.

And what should be removed in th database to remove this

?

This is visible after I set has_actions = false. And click on the x or o show 404.

lunny · 2024-10-20T20:00:31Z

I think these commit status can be removed manually from the database with

delete from commit_status where repo_id = (select id from repository where is_mirror = true)
delete from commit_status_summary where repo_id = (select id from repository where is_mirror = true)

somera · 2024-10-21T11:23:39Z

I this case I should send only has_actions = false as an json? Right?

Yes. You only need to set the fields that need to be changed.

I have now a script which I run after I added some new mirrors.

But for the next release there will be a fix in gitea? Means, when I import new mirror "Actions" will be deactivated by default?

I think these commit status can be removed manually from the database with

delete from commit_status where repo_id = (select id from repository where is_mirror = true)
delete from commit_status_summary where repo_id = (select id from repository where is_mirror = true)

Thx. I try this next weekend with an Gitea-DB backup.

somera · 2024-10-24T07:45:32Z

@lunny is there a fix in the next release that for new mirrors "Actions" will be not activated?

lunny · 2024-10-25T01:40:02Z

Yes, I think there should be a PR to fix that but there is no ETA

somera added the type/bug label Oct 10, 2024

Zettat123 added the topic/gitea-actions related to the actions of Gitea label Oct 14, 2024

lunny added the issue/workaround it is or has a workaround label Oct 22, 2024

Zettat123 linked a pull request Nov 5, 2024 that will close this issue

Add a doctor check to disable the "Actions" unit for mirrors #32424

Open

Strange action errors in mirrored projects?! #32232

Strange action errors in mirrored projects?! #32232

Comments

somera commented Oct 10, 2024 • edited Loading

Description

Gitea Version

Can you reproduce the bug on the Gitea demo site?

Log Gist

Screenshots

Git Version

Operating System

How are you running Gitea?

Database

somera commented Oct 10, 2024 • edited Loading

somera commented Oct 10, 2024

somera commented Oct 10, 2024

lunny commented Oct 10, 2024

somera commented Oct 12, 2024

lunny commented Oct 13, 2024

Zettat123 commented Oct 14, 2024

somera commented Oct 14, 2024 • edited Loading

somera commented Oct 14, 2024 • edited Loading

somera commented Oct 14, 2024 • edited Loading

somera commented Oct 14, 2024

Zettat123 commented Oct 15, 2024 • edited Loading

yp05327 commented Oct 15, 2024 • edited Loading

Zettat123 commented Oct 15, 2024

yp05327 commented Oct 15, 2024 • edited Loading

somera commented Oct 15, 2024

somera commented Oct 15, 2024 • edited Loading

yp05327 commented Oct 15, 2024 • edited Loading

yp05327 commented Oct 15, 2024

Zettat123 commented Oct 15, 2024

somera commented Oct 15, 2024

yp05327 commented Oct 15, 2024 • edited Loading

Zettat123 commented Oct 15, 2024

somera commented Oct 15, 2024 • edited Loading

somera commented Oct 15, 2024 • edited Loading

somera commented Oct 15, 2024

somera commented Oct 15, 2024

yp05327 commented Oct 16, 2024

somera commented Oct 17, 2024

yp05327 commented Oct 17, 2024

somera commented Oct 17, 2024

Zettat123 commented Oct 18, 2024 • edited Loading

somera commented Oct 18, 2024

lunny commented Oct 20, 2024

somera commented Oct 21, 2024

somera commented Oct 24, 2024

lunny commented Oct 25, 2024

somera commented Oct 10, 2024 •

edited

Loading

somera commented Oct 10, 2024 •

edited

Loading

somera commented Oct 14, 2024 •

edited

Loading

somera commented Oct 14, 2024 •

edited

Loading

somera commented Oct 14, 2024 •

edited

Loading

Zettat123 commented Oct 15, 2024 •

edited

Loading

yp05327 commented Oct 15, 2024 •

edited

Loading

yp05327 commented Oct 15, 2024 •

edited

Loading

somera commented Oct 15, 2024 •

edited

Loading

yp05327 commented Oct 15, 2024 •

edited

Loading

yp05327 commented Oct 15, 2024 •

edited

Loading

somera commented Oct 15, 2024 •

edited

Loading

somera commented Oct 15, 2024 •

edited

Loading

Zettat123 commented Oct 18, 2024 •

edited

Loading